SMART: Scalable, Bandwidth-Aware Monitoring of Continuous Aggregation Queries

We present SMART, a scalable, bandwidth-aware monitoring system that maximizes result precision of continuous aggregate queries over distributed data streams. While previous approaches reduce bandwidth cost under fixed precision constraints, in practice, monitoring systems may still incur a substantial cost risking overload under bursty traffic conditions. SMART therefore bounds the worst-case system cost to provide overload resilience and to facilitate practical deployment of monitoring systems. The primary challenge for SMART is how to select dynamic updates at each node in a distributed system to maximize global precision while keeping per-node monitoring bandwidth below a specified budget. To address this challenge, SMART’s hierarchical algorithm (1) allocates bandwidth budgets in a nearoptimal manner to maximize global precision and (2) selftunes bandwidth settings to improve precision under dynamic workloads. Our prototype implementation of SMART provides key solutions to (a) prioritize pending updates for multi-attribute queries, (b) build bounded fan-in, load-aware aggregation trees to improve accuracy and fast anomaly detection, and (c) combine temporal batching with arithmetic filtering to reduce load and to quantify result staleness. Finally, our evaluation using simulations and a network monitoring application shows that SMART improves accuracy by up to an order of magnitude compared to uniform bandwidth allocation and performs close to the optimal algorithm under modest bandwidth budgets.